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(1) Real Party in Interest 

A statement identifying by name the real party in interest is contained in the brief. 

(2) Related Appeals and Interferences 

The examiner is not aware of any related appeals, interferences, or judicial 
proceedings which will directly affect or be directly affected by or have a bearing on the 
Board's decision in the pending appeal. 

(3) Status of Claims 

The statement of the status of claims contained in the brief is correct. 

(4) Status of Amendments After Final 

The appellant's statement of the status of amendments after final rejection 
contained in the brief is correct. 

(5) Summary of Claimed Subject Matter 

The summary of claimed subject matter contained in the brief is correct. 

(6) Grounds of Rejection to be Reviewed on Appeal 

The appellant's statement of the grounds of rejection to be reviewed on appeal is 
correct. 

(7) Claims Appendix 

The copy of the appealed claims contained in the Appendix to the brief is correct. 
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(8) Evidence Relied Upon 



20030212679 



VEBKALAYA 



11-2003 



6,513,025 



ROSEN 



1-2003 



(9) Grounds of Rejection 

The following ground(s) of rejection are applicable to the appealed claims: 
Claims 1-35 are rejected under 35 U.S.C. 103(a) as being unpatentable over 

Venkayala et al. (US Pub. No 20030212679), in view of Rosen et al. (US Patent No. 

6,513,025). 

As to claims 1, 28,Venkayala teaches a method/program storage device/system of 
searching data in databases using an ensemble of models (i.e. seeds models, [0018]), said method 
program storage device/system comprising: 

ordering models within said ensemble in order of prediction accuracy (i.e. Trained model 
1 10 may also be evaluated and adjusted in order to improve the quality, i.e. prediction accuracy, 
of the model, [0019]), with the most accurate model being first in said order (i.e. a topmost 
category including a class value having a highest associated probability, [0010]), ([0008-0009, 
0018,0019, 0035]); 

selecting a sub-ensemble of said models that meets a given level of confidence (i.e. how 
much confidence may be placed in the prediction, [0023], the selected class values are those 
meeting the selection criteria presented in prediction parameters, [0024]), wherein said sub- 
ensemble in said order prediction accuracy ([0010, 0035]), (i.e. the selection criteria may include 
a limit on the number of class values that are to be selected, or may indicate that the class values 
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are to be sorted based on their associated probabilities. The selected class values, which are 
included in multi-category apply output 122, are output from step 120, [0024]); and 

applying said sub-ensemble, in place of said ensemble, to an example to make a 
prediction (i.e. The selected class values, which are included in multi-category apply output, 
[0024]) ([0010, 0035]), (i.e. generate one or more scores for each row of data in scoring data 
116. The scores for each row of data indicate how closely the row of data matches attributes of 
the model, how much confidence may be placed in the prediction, how likely each output 
prediction/recommendation to be true, and other statistical indicators. Scored data 118 is output 
from scoring step 112 and includes predictions or recommendations, along with corresponding 
probabilities for the scored data, [0023]). 

Venkayala teaches the step of forming sub-ensembles, wherein said sub-ensembles in 
said order of prediction accuracy (i.e. model apply, [0017]). 

Venkayala does not expressly teach "models are joined together in said sub-ensemble". 
However, Rosen teaches "models are joined together in said sub-ensemble" (i.e. The most 
dependable classification model, that is the classification model associated with the 
dependability model indicating the highest dependability for the unlabeled example, is chosen to 
classify each unclassified example (col. 4, line 66 to col. 5, line 9). 

It would have been obvious to one of ordinary skill of the art having the teaching of 
Venkayala and Rosen at the time the invention was made to modify the system of Venkayala to 
include wherein models are joined together in said sub-ensemble in said order of prediction 
accuracy as taught by Rosen. One of ordinary skill in the art would be motivated to make this 
combination in order to select the most appropriate classification model, and the prediction of 
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that classification model is then accepted in view of Rosen, as doing so would give the added 
benefit of coalescing the predictions of the base models by learning the relationships between 
those predictions and the correct prediction, identifying a specific classification model as the one 
responsible for producing a final prediction and to including a simple explanation of why the 
prediction was made, and revising the selected model as the models change over time as taught 
by Rosen (col. 2, lines 31-42). 

As per claim 8, Venkayala teaches a method of searching data in databases using an 
ensemble of models (i.e. seeds models, [0018]), said method comprising: 

ordering models within said ensemble in order of prediction accuracy (i.e. Trained model 
110 may also be evaluated and adjusted in order to improve the quality, i.e. prediction accuracy 
of the model, [0019]), with the most accurate model being first in said order (i.e. a topmost 
category including a class value having a highest associated probability, [0010]) ([0008-0009, 
0018,0019, 0035]); 

selecting a sub-ensemble of said models that meets a given level of confidence (i.e. how 
much confidence may be placed in the prediction, [0023], the selected class values are those 
meeting the selection criteria presented in prediction parameters, [0024]), wherein said sub- 
ensemble in said order prediction accuracy, such that said sub-ensemble include only the most 
accurate models (i.e. a topmost category including a class value having a highest associated 
probability, [0010]) ([0008, 0009, 0035]), (i.e. the selection criteria may include a limit on the 
number of class values that are to be selected, or may indicate that the class values are to be 
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sorted based on their associated probabilities. The selected class values, which are included in 
multi-category apply output 122, are output from step 120, [0024]); and 

applying said sub-ensemble, in place of said ensemble, to an example to make a 
prediction (i.e. The selected class values, which are included in multi-category apply output, 
[0024]) ([0010, 0035]), (i.e. generate one or more scores for each row of data in scoring data 
116. The scores for each row of data indicate how closely the row of data matches attributes of 
the model, how much confidence may be placed in the prediction, how likely each output 
prediction/recommendation to be true, and other statistical indicators. Scored data 118 is output 
from scoring step 112 and includes predictions or recommendations, along with corresponding 
probabilities for the scored data, [0023]). 

Venkayala teaches the step of forming sub-ensembles, wherein said sub-ensembles in 
said order of prediction accuracy (i.e. model apply, [0017]). 

Venkayala does not expressly teach "models are joined together in said sub-ensemble". 

However, Rosen teaches "models are joined together in said sub-ensemble" (i.e. The 
most dependable classification model, that is the classification model associated with the 
dependability model indicating the highest dependability for the unlabeled example, is chosen to 
classify each unclassified example (col. 4, line 66 to col. 5, line 9). 

It would have been obvious to one of ordinary skill of the art having the teaching of 
Venkayala and Rosen at the time the invention was made to modify the system of Venkayala to 
include wherein models are joined together in said sub-ensemble in said order of prediction 
accuracy as taught by Rosen. One of ordinary skill in the art would be motivated to make this 
combination in order to select the most appropriate classification model, and the prediction of 
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that classification model is then accepted in view of Rosen, as doing so would give the added 
benefit of coalescing the predictions of the base models by learning the relationships between 
those predictions and the correct prediction, identifying a specific classification model as the one 
responsible for producing a final prediction and to including a simple explanation of why the 
prediction was made, and revising the selected model as the models change over time as taught 
by Rosen (col. 2, lines 31-42). 

As per claim 15, Venkayala teaches a method of searching data in databases using an 
ensemble of models (i.e. seeds models, [0018]), said method comprising: 

performing training (i.e. training/model building, [0017, 0019]) comprising: 
ordering models within said ensemble in order of prediction accuracy (i.e. Trained model 
110 may also be evaluated and adjusted in order to improve the quality, i.e. prediction accuracy, 
of the model, [0019]), with the most accurate model being first in said order (i.e. a topmost 
category including a class value having a highest associated probability, [0010]) ([0008-0009, 
0018,0019, 0035]); 

forming sub-ensembles (i.e. model apply, [0017]), wherein said sub-ensemble in said 
order of prediction accuracy ([0010, 0035]); 

calculating confidence values of each of said sub-ensembles (i.e. to generate one or more 
scores for each row of data in scoring data. The scores for each row of data indicate how closely 
the row of data matches attributes of the model, how much confidence may be placed in the 
prediction, [0023]); and making a prediction comprising: 
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selecting a sub-ensemble of said models that meets a given level of confidence (i.e. The 
selected class values are those meeting the selection criteria presented in prediction parameters, 
[0024]) ([0010, 0035]), (i.e. the selection criteria may include a limit on the number of class 
values that are to be selected, or may indicate that the class values are to be sorted based on 
their associated probabilities. The selected class values, which are included in multi-category 
apply output 122, are output from step 120, [0024]); and 

applying said sub-ensemble, in place of said ensemble, to an example to make a 
prediction (i.e. The selected class values, which are included in multi-category apply output, 
[0024]) ([0010, 0035]), (i.e. generate one or more scores for each row of data in scoring data 
116. The scores for each row of data indicate how closely the row of data matches attributes of 
the model, how much confidence may be placed in the prediction, how likely each output 
prediction/recommendation to be true, and other statistical indicators. Scored data 118 is output 
from scoring step 112 and includes predictions or recommendations, along with corresponding 
probabilities for the scored data, [0023]). 

Venkayala teaches the step of forming sub-ensembles, wherein said sub-ensembles in 
said order of prediction accuracy (i.e. model apply, [0017]). 

Venkayala does not expressly teach "joining different number of models together to form 
sub-ensembles". 

However, Rosen teaches "joining different number of models together to form sub- 
ensembles" (i.e. The most dependable classification model, that is the classification model 
associated with the dependability model indicating the highest dependability for the unlabeled 
example, is chosen to classify each unclassified example (col. 4, line 66 to col. 5, line 9). 
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It would have been obvious to one of ordinary skill of the art having the teaching of 
Venkayala and Rosen at the time the invention was made to modify the system of Venkayala to 
include joining different numbers of models together to form sub-ensembles, wherein models are 
joined together in said sub-ensemble in said order of prediction accuracy as taught by Rosen. 
One of ordinary skill in the art would be motivated to make this combination in order to select 
the most appropriate classification model, and the prediction of that classification model is then 
accepted in view of Rosen, as doing so would give the added benefit of coalescing the 
predictions of the base models by learning the relationships between those predictions and the 
correct prediction, identifying a specific classification model as the one responsible for 
producing a final prediction and to including a simple explanation of why the prediction was 
made, and revising the selected model as the models change over time as taught by Rosen (col. 2, 
lines 31-42). 

As per claim 21, Venkayala teaches a service of searching data in databases using an 
ensemble of models (i.e. seeds models, [0018]), said service comprising: 

ordering models within said ensemble in order of prediction accuracy (i.e. Trained model 
110 may also be evaluated and adjusted in order to improve the quality, i.e. prediction accuracy, 
of the model, [0019]), with the most accurate model being first in said order (i.e. a topmost 
category including a class value having a highest associated probability, [0010]) ([0008-0009, 
0018,0019, 0035]); 

selecting a sub-ensemble of said models that meets a given level of confidence (i.e. how 
much confidence may be placed in the prediction, [0023], the selected class values are those 
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meeting the selection criteria presented in prediction parameters, [0024]), wherein said sub- 
ensemble in said order prediction accuracy ([0010, 0035]), (i.e. the selection criteria may include 
a limit on the number of class values that are to be selected, or may indicate that the class values 
are to be sorted based on their associated probabilities. The selected class values, which are 
included in multi-category apply output 122, are output from step 120, [0024]); and 

applying said sub-ensemble, in place of said ensemble, to an example to make a 
prediction (i.e. The selected class values, which are included in multi-category apply output, 
[0024]) ([0010, 0035]), (i.e. generate one or more scores for each row of data in scoring data 
116. The scores for each row of data indicate how closely the row of data matches attributes of 
the model, how much confidence may be placed in the prediction, how likely each output 
prediction/recommendation to be true, and other statistical indicators. Scored data 118 is output 
from scoring step 112 and includes predictions or recommendations, along with corresponding 
probabilities for the scored data, [0023]). 

Venkayala teaches the step of forming sub-ensembles, wherein said sub-ensembles in 
said order of prediction accuracy (i.e. model apply, [0017]). 

Venkayala does not expressly teach "models are joined together in said sub-ensemble". 

However, Rosen teaches "models are joined together in said sub-ensemble" (i.e. The 
most dependable classification model, that is the classification model associated with the 
dependability model indicating the highest dependability for the unlabeled example, is chosen to 
classify each unclassified example (col. 4, line 66 to col. 5, line 9). 

It would have been obvious to one of ordinary skill of the art having the teaching of 
Venkayala and Rosen at the time the invention was made to modify the system of Venkayala to 
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include wherein models are joined together in said sub-ensemble in said order of prediction 
accuracy as taught by Rosen. One of ordinary skill in the art would be motivated to make this 
combination in order to select the most appropriate classification model, and the prediction of 
that classification model is then accepted in view of Rosen, as doing so would give the added 
benefit of coalescing the predictions of the base models by learning the relationships between 
those predictions and the correct prediction, identifying a specific classification model as the one 
responsible for producing a final prediction and to including a simple explanation of why the 
prediction was made, and revising the selected model as the models change over time as taught 
by Rosen (col. 2, lines 31-42). 

As per claim 35, Venkayala teaches a system of searching data in databases using an 
ensemble of models (i.e. seeds models, [0018]), said system comprising: 

means for ordering models within said ensemble in order of prediction accuracy (i.e. 
Trained model 110 may also be evaluated and adjusted in order to improve the quality, i.e. 
prediction accuracy, of the model, [0019]), with the most accurate model being first in said order 
(i.e. a topmost category including a class value having a highest associated probability, [0010]), 
([0008-0009, 0018, 0019, 0035]); 

means for selecting a sub-ensemble of said models that meets a given level of confidence 
(i.e. how much confidence may be placed in the prediction, [0023], the selected class values are 
those meeting the selection criteria presented in prediction parameters, [0024]), wherein said 
sub-ensemble in said order prediction accuracy ([0010, 0035]), (i.e. the selection criteria may 
include a limit on the number of class values that are to be selected, or may indicate that the 
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class values are to be sorted based on their associated probabilities. The selected class values, 
which are included in multi-category apply output 122, are output from step 120, [0024]); and 

means for applying said sub-ensemble, in place of said ensemble, to an example to make 
a prediction (i.e. The selected class values, which are included in multi-category apply output, 
[0024]) ([0010, 0035]), (i.e. generate one or more scores for each row of data in scoring data 
116. The scores for each row of data indicate how closely the row of data matches attributes of 
the model, how much confidence may be placed in the prediction, how likely each output 
prediction/recommendation to be true, and other statistical indicators. Scored data 118 is output 
from scoring step 112 and includes predictions or recommendations, along with corresponding 
probabilities for the scored data, [0023]). 

Venkayala teaches the step of forming sub-ensembles, wherein said sub-ensembles in 
said order of prediction accuracy (i.e. model apply, [0017]). 

Venkayala does not expressly teach "models are joined together in said sub-ensemble". 
However, Rosen teaches "models are joined together in said sub-ensemble" (i.e. The most 
dependable classification model, that is the classification model associated with the 
dependability model indicating the highest dependability for the unlabeled example, is chosen to 
classify each unclassified example (col. 4, line 66 to col. 5, line 9). 

It would have been obvious to one of ordinary skill of the art having the teaching of 
Venkayala and Rosen at the time the invention was made to modify the system of Venkayala to 
include wherein models are joined together in said sub-ensemble in said order of prediction 
accuracy as taught by Rosen. One of ordinary skill in the art would be motivated to make this 
combination in order to select the most appropriate classification model, and the prediction of 
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that classification model is then accepted in view of Rosen, as doing so would give the added 
benefit of coalescing the predictions of the base models by learning the relationships between 
those predictions and the correct prediction, identifying a specific classification model as the one 
responsible for producing a final prediction and to including a simple explanation of why the 
prediction was made, and revising the selected model as the models change over time as taught 
by Rosen (col. 2, lines 31-42). 

As to claims 2, 9, 16, 22, 29, Venkayala teaches said sub-ensemble includes fewer 
models than said ensemble (i.e. The selection criterion may comprise one of a topmost category 
including a class value having a highest associated probability, top N categories including N 
class values having highest associated probabilities, bottom N categories including N class 
values having lowest associated probabilities, or a set of select class values specified by the user 
and their associated probabilities and ranks, [0010]). 

As to claims 3, 10, 17, 23, 30, Venkayala teaches said confidence is a measure of how 
closely results from said sub-ensemble will match results from said ensemble (i.e. The scores for 
each row of data indicate how closely the row of data matches attributes of the model, how much 
confidence may be placed in the prediction, how likely each output, [0049]). 

As to claims 4, 11, 18, 24, 31, Venkayala teaches the size of each sub-ensemble is 
different and has a potentially different level of confidence (i.e. The selection criterion may 
comprise one of a topmost category including a class value having a highest associated 



Application/Control Number: 10/737,123 Page 14 

Art Unit: 2159 

probability, top N categories including N class values having highest associated probabilities, 
bottom N categories including N class values having lowest associated probabilities, or a set of 
select class values specified by the user and their associated probabilities and ranks, [0010]). 

As to claims 5, 12, 19, 25, 32, Venkayala teaches the size of said ensemble is fixed (i.e. 
the selection criteria may include a limit on the number of class values that are to be selected, 
[0050]), (i.e. a predetermined set of categories, [0020]). 

As to claims 6, 13, 20, 26, 33, Venkayala teaches as the level of confidence is raised, a 
sub-ensemble that has more models will be selected in said selecting process (i.e. The selection 
criteria may be defined by desired results data, [0024]), and as the level of confidence is lowered, 
a sub-ensemble that has fewer models will be selected in said selecting process (i.e. The 
selection criterion may comprise one of a topmost category including a class value having a 
highest associated probability, top N categories including N class values having highest 
associated probabilities, bottom N categories including N class values having lowest associated 
probabilities, or a set of select class values specified by the user and their associated probabilities 
and ranks , [0010]). 

As to claims 7, 14, 27, 34, Venkayala teaches before said selecting, calculating 
confidence values of different sub-ensembles (i.e. to generate one or more scores for each row of 
data in scoring data. The scores for each row of data indicate how closely the row of data 
matches attributes of the model, how much confidence may be placed in the prediction, [0049]). 
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(10) Response to Argument 

1 . Overview of the claimed invention and reference Venkayala 

In summary, the instant application relates to a classifier ensemble method in the 
machine-learning and data-mining environment where the use of multiple models 
(ensembles) can scale up data mining over very large databases and datasets. 
Ensembles of models (classifiers) achieve the same or even better accuracy than a 
single model computed from the entire dataset. This invention provides an adaptive 
pruning the ensemble approach to reduce the "expected" number of classifiers 
employed in prediction by generating diverse (i.e. wide range) classifiers, and selecting 
the proper base classifier to join the ensemble. It measures the confidence of a 
prediction by a subset of classifiers in the ensemble. Thus, confidence is used to decide 
if more classifiers are needed in order to produce a prediction that is the same as the 
original ensemble with more classifiers. The areas of applications that benefit from this 
invention include fraud detection, risk management, trading surveillances, medical 
diagnosis, intrusion detection, as well as security and exchange ([0007], [0017]). 

Similarly, Venkayala discloses a supervised learning is a collection of techniques 
in data mining that are used to build a model from a given set of records, known as a 
training set, whose class values are known a priori. Once the model is built, it is tested 
against another set of records with known class values, known as a test set, in order to 
quantify the quality of the model. It is then used to predict (or score) unknown class 
values of real-world records. This last stage where the model is used for prediction is 
termed apply [0003]. This technique can also be used for unsupervised models such as 
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clustering models. Clustering analysis identifies clusters embedded in the data where a 
cluster is a collection of records in the data that are similar to one another. Once 
clusters are identified from a given set of records, one can get predictions for new 
records on which cluster each record is likely to belong. Such predictions may be 
associated with probability, the quality of fit, which describes how well a given record fits 
in the predicted cluster, and the distance from the center of the predicted cluster [0005]. 
Analogously to the instant application, Venkayala discloses that the traditional 
applications of such supervised learning techniques include retail target marketing, 
medical diagnosis, weather prediction, credit approval, customer segmentation, and 
fraud detection [0003]. 

It is noted that as specified in the instant specification, a model is a classifier, and 
an ensemble of models is group of classifiers, (See [0007, 0017], Specification). 

Based on the similarities of the inventions, the following terms of the instant 
application read on Venkayala's as: 

searching data in databases using an ensemble of models equates to the 
step of providing a multi-category apply operation in a data mining system that produces 
output with multiple class values and their associated probabilities (Venkayala ([0006]). 

an ensemble of models equates to the group of categories of Venkayala. 

It should be noted that a category of Venkayala includes a class value, thus a set 
of selected class values of Venkayala equates to a sub-ensemble of Applicants. 

performing training orders models within the ensemble in order of 
prediction accuracy, with the most accurate model being first ([0005]) equates to 
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"model building (supervised learning or unsupervised) and scoring of models (model 
apply), is shown in FIG. 1. The training/model building step 102 involves generating the 
models that are used to perform data mining recommendation and prediction" 
(Venkayala [0017]). 

the sub-ensembles include fewer models than the ensemble and each sub- 
ensemble includes only the most accurate models of Applicants ([0005]) equates to 
top N categories including N class values having highest associated probabilities, 
bottom N categories including N class values having lowest associated probabilities, or 
a set of select class values specified by the user and their associated probabilities and 
ranks (Venkayala [0008]). 

It should be noted that Venkayala teaches a selected class values corresponding 
to categories (i.e. ensemble of models); hence, the set of selected class values 
corresponds to a sub set categories, e.g. top N categories including N class values (i.e. 
the selection criteria may include a limit on the number of class values that are to be 
selected, or may indicate that the class values are to be sorted based on their 
associated probabilities. The selected class values, which are included in multi-category 
apply output 122, are output from step 120, [0024]). Therefore, a sub set categories of 
Venkayala equates to a sub-ensemble of Applicants. 

calculating confidence values of each of the sub-ensembles equates to how 
much confidence may be placed in the prediction ([0023]), the selected class values 
are those meeting the selection criteria presented in prediction parameters 114 ([0024]). 
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the models are joined together in the sub-ensemble in the order of 
prediction accuracy, although Venkayala does not state the term "join", this limitation 
is implicitly taught by Venkayala as a set of selected class values including in multi- 
category ([0024]). 

the confidence is a measure of how closely results from the sub-ensemble 
will match results from the ensemble equates to The scores for each row of data 
indicate how closely the row of data matches attributes of the model, how much 
confidence may be placed in the prediction ([0023]). 

the size of each of the sub-ensembles is variable depending upon the level 
of confidence equates a set of select class values specified by the user and their 
associated probabilities and ranks (Venkayala [0008]). 

Examiner's Position: 

a. Independent Claims 1, 8, 15, 21, 28, 35 

The proposed combination of Venkayala teaches/suggests the claimed features 
"applying said sub ensemble, in place of said ensemble, to an example to make a 
prediction" as "generating one or more scores for each row of data in scoring data 116. 
The scores for each row of data indicate how closely the row of data matches attributes 
of the model, how much confidence may be placed in the prediction, how likely each 
output prediction/recommendation to be true, and other statistical indicators. Scored 
data 118 is output from scoring step 112 and includes predictions or recommendations, 
along with corresponding probabilities for the scored data", [0023]. 
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Appellant contends that "nothing within Venkayala discloses ensembles and sub- 
ensembles", however, as explained in the preceding paragraphs, ensembles are the 
groups of categories. In addition, since each category of Venkayala comprises a class 
value, a set of selected class value is thus corresponds to the claimed feature sub- 
ensemble. 

Furthermore, contrary to Appellant's belief that only a single model is built and 
utilized, as shown in Fig. 1 , the Examiner respectfully points out that models are built 
(i.e. multi- category apply output) in accordance with seed models in combination with 
training parameter and training data. As taught by Venkayala, "this technique can also 
be used for unsupervised models such as clustering models", ([0005]). 

Appellant concedes that "nothing within Venkayala teaches or suggests the 
creation of ensembles and sub-ensembles of models to make a prediction", the 
Examiner respectfully traverses. 

It is noted that Venkayala teaches "a method for multi-category apply in a data 
mining system that comprises the steps of receiving input data for scoring including a 
plurality of rows of data applied to a data mining model and generating multi- 
category apply output with a plurality of predicted class values and their 
associated probabilities based on the received input data and a selection 
criterion" [0007]; therefore, the step of receiving input data for scoring including a 
plurality of rows of data applied to a data mining model illustrates how rows of data 
reads on an example of the claimed limitation. 
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Venkayala teaches that receiving input data for scoring including a plurality of 
rows of data applied to a data mining model and generating multi-category apply output 
with a plurality of predicted class values and their associated probabilities based on the 
received input data and a selection criterion where generating multi-category apply 
output may comprise the steps of generating input data tables including active attributes 
and source attributes, evaluating probabilities of categories of a target attribute to 
determine those meeting the selection criterion, and generating an output data table 
including a plurality of class values of the target attribute and their associated 
probabilities, the selected class values having probabilities meeting the selection 
criterion, ( [0007]); thus, applying sub-ensemble equates to the step of the selection 
criterion or the selected class values. 

It is also noted that the "selection criterion" of Venkayala is a set of class values 
or top N categories, which can also be equated to the sub-ensemble (i.e. top N 
categories including N class values having highest associated probabilities, bottom N 
categories including N class values having lowest associated probabilities, or a set of 
select class values specified by the user and their associated probabilities and ranks, 
See [0009], Venkayala). Further, the step of generating multi-category apply output with 
a plurality of predicted class values and their associated probabilities based on the 
received input data and a selection criterion, [0007]) indicates how generating multi- 
category of Venkayala reads on the claimed limitation make a prediction, it is the 
Examiner's position that contrary to Appellant's belief, the selection criterion, which is a 
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set of selected class values that can also be equated to the sub-ensemble, is used in 
place of the model to make predictions. 



b. Dependent Claims 2, 9, 16, 22, 29 

As detailed in the preceding paragraphs, ensembles are the groups of 
categories; and since each category of Venkayala comprises a selected class value, the 
set of selected class value is thus corresponds to the claimed feature sub-ensemble. 
Furthermore, contrary to Appellant's belief that only a single model is built and utilized; 
however, in view of Fig. 1 , the Examiner respectfully points out that models are built (i.e. 
multi- category apply output) in accordance with seed models in combination with 
training parameter and training data. As taught by Venkayala, "this technique can also 
be used for unsupervised models such as clustering models", ([0005]), and "model 
building (supervised learning or unsupervised) and scoring of models (model apply), is 
shown in FIG. 1. The training/model building step 102 involves generating the models 
that are used to perform data mining recommendation and prediction", [001 7]. 

Thus, the proposed combination of Venkayala teaches/suggests the claimed 
features "sub-ensemble includes fewer models than said ensemble" as Venkayala 
teaches top N categories including N class values having highest associated 
probabilities, bottom N categories including N class values having lowest associated 
probabilities, or a set of selected class values specified by the user and their associated 
probabilities and ranks . [0008]. 



Application/Control Number: 10/737,123 Page 22 

Art Unit: 2159 

c. Dependent Claims 3, 10, 17, 23, 30 

The proposed combination of Venkayala teaches/suggests the claimed features 
"wherein said confidence is a measure of how closely results from said sub-ensemble 
will match results from said ensemble" as "The scores for each row of data indicate how 
closely the row of data matches attributes of the model, how much confidence may be 
placed in the prediction, how likely each output prediction/recommendation to be true, 
and other statistical indicators. Scored data 118 is output from scoring step 112 and 
includes predictions or recommendations, along with corresponding probabilities for the 
scored data", [0023]. 

According to the instant specification, "the invention is applicable to a wide range 
of ensembles. It measures the confidence of a prediction by a subset of classifiers in the 
ensemble . Thus, confidence is used to decide if more classifiers are needed in order to 
produce a prediction that is the same as the original ensemble with more classifiers " 
[0017], and " the prediction accuracy for each model is calculated bv checking 
predictions made with each model separately using validation data (or training data that 
was initially used to train the model, if insufficient validation data is unavailable), [0018]; 
therefore, "row of data" corresponds to "an example", or validation data or training data 
to train a model, an ensemble, or a sub-ensemble (i.e. a set of select class value 
specified by the user and their associated probabilities and ranks) as "rows of data 
applied to a data mining model and associated measures, including probabilities of 
predictions, ranks, quality of fit, or distance" (see claim 1 , Venkayala); thus the scores 
for each row is a level of confidence to measure how closely results from row of data in 
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accordance with selected class value (sub-ensemble) will match results from models, or 
set of categories, multi-category (ensemble). 

d. Dependent Claims 6, 13, 20, 26, 33 

The proposed combination of Venkayala teaches/suggests the claimed features 
"wherein as the level of confidence is raised, a sub-ensemble that has more models will 
be selected in said selecting process, and as the level of confidence is lowered, a sub- 
ensemble that has fewer models will be selected in said selecting process" as "the 
selection criterion may comprise one of a topmost category including a class value 
having a highest associated probability, top N categories including N class values 
having highest associated probabilities, bottom N categories including N class values 
having lowest associated probabilities, or a set of select class values specified by the 
user and their associated probabilities and ranks ", ([0010]). 

Venkayala also read on the claimed limitations as "Scored data 118 is input to 
multi-category apply step, which evaluates the probabilities associated with the class 
values and selects a plurality of class values and their associated measures, such as 
probabilities . The selected class values are those meeting the selection criteria 
presented in prediction parameters 1 14. The selection criteria may be defined by 
desired results data and/or by predefined or default criteria included in 
selection/generation step 120 . In addition, the selection criteria may include a limit on 
the number of class values that are to be selected, or may indicate that the class values 
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are to be sorted based on their associated probabilities . The selected class values, 
which are included in multi-category apply output 122, are output from step 120, [0024]. 



As discussed above, the proposed combination of Venkayala and Rosen render 
the independent claims 1 , 8, 1 5, 21 , 28 and dependent claims 4, 5, 7, 1 1 , 1 2, 1 4, 1 8, 1 9, 
24,25, 27, 31,32, 34 obvious. 

Based on the foregoing discussion, it is submitted that all claims are not 
patentably distinct over the cited art of record. 

(11) Related Proceeding(s) Appendix 

No decision rendered by a court or the Board is identified by the examiner in the 
Related Appeals and Interferences section of this examiner's answer. 

For the above reasons, it is believed that the rejections should be sustained. 
Respectfully submitted, 
/Miranda Le/ 

Primary Examiner, Art Unit 2159 



e. Dependent Claims 4, 5, 7, 11, 12, 14, 18, 19, 24, 25, 27 
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Supervisory Patent Examiner, Art Unit 
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